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September 30, 2009 

Dr. Eric Smith 
Commissioner of Education 
Florida Department of Education 
325 West Gaines Street 
Tallahassee, Florida 32399-0400 

Dear Dr. Smith: 

Enclosed is our final audit report, Control Number ED-OIG/A04I0043, entitled Florida Department of 
Education Controls Over State Assessment Scoring. This report incorporates the comments you provided 
in response to the draft report. If you have any additional comments or information that you believe may 
have a bearing on the resolution of this audit, you should send them directly to the following Education 
Department officials], who will consider them before taking final Departmental action on this audit: 

Thelma Melendez de Santa Ana 
Assistant Secretary 

Office of Elementary and Secondary Education 
400 Maryland Avenue, SW 
Room 3W315 
Washington, D.C. 20202 

It is the policy of the U. S. Department of Education to expedite the resolution of audits by initiating 
timely action on the findings and recommendations contained therein. Therefore, receipt of your 
comments within 30 days would be appreciated. 

In accordance with the Freedom of Information Act (5 U.S.C. § 552), reports issued by the Office of 
Inspector General are available to members of the press and general public to the extent information 
contained therein is not subject to exemptions in the Act. 

Sincerely, 
/s/ 

Denise M. Wempe 

Regional Inspector General for Audit 
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EXECUTIVE SUMMARY 



The objective of our audit was to determine whether controls over scoring of assessments at the 
Florida Department of Education (FLDOE) were adequate to provide reasonable assurance that 
assessment results are reliable. Our review covered the Florida Comprehensive Assessment Test 
(FCAT)® administered in school year 2007-2008. 1 The FCAT is used for evaluating individual 
students and making adequate yearly progress (AYP) determinations under Section 1111(b)(2) of the 
Elementary and Secondary Education Act (ESEA), as amended by the No Child Left Behind Act of 
2001. Section 1111(b)(3) of the ESEA also requires accurate, reliable, high-quality assessment data. 
Assessments are used to hold schools accountable for student achievement. 

We found that FLDOE has internal controls over scoring the FCAT assessment to provide reasonable 
assurance that assessment results are reliable. However, we found discrepancies in the FCAT gridded 
responses and that FLDOE did not sufficiently monitor contractor activities to ensure compliance with 
contract requirements. In addition, our audit was delayed because FLDOE' s contractor limited access 
to documentation required for our audit. 

Based on our findings, we recommend that the Assistant Secretary of the Office of Elementary and 
Secondary Education require FLDOE to 

1 . 1 Ensure the contractor is correctly setting the Technology Intensity Calibration Algorithm 
to capture students' gridded responses in the scanner. For responses that are manually 
entered, have a second verification of the entry to ensure the gridded response items are 
captured correctly. 

1.2 Implement procedures to test a sample of the gridded responses during live scoring to 
ensure students' gridded responses are accurately scanned; 

2. 1 Use unique identifiers instead of name, social security numbers, and dates of birth on 
assessment documents. 

2.2 Ensure that all contractors are aware of the proper handling of PII and include language 
in their contracts to properly address the correct handling procedures related to the 
disposal of student PII. 

2.3 Monitor the contractor to ensure compliance with contract provisions and include a table 
of penalties in the contract for non-compliance with contractual requirements. 

2.4 Monitor document control procedures at the contractor facilities at least annually. 

3.1 Include a Federal audit clause provision in contracts for Department funded programs. 

3.2 Include a table of penalties in the contract for non-compliance with a Federal audit. 



1 The Spring 2008 FCAT are the assessments administered during the 2007-2008 school year used to calculate adequate 
yearly progress. 
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In its comments to the draft report, FLDOE did not agree with Finding No. 1 and Finding No. 2. 
FLDOE agreed in part with Finding No. 3 but disagreed with the part of the finding related to limiting 
access to documentation. FLDOE provided corrective actions to address Recommendations 1.1, 2.2, 
2.3, 2.4, and 3.1. FLDOE provided corrective actions that partially address Recommendation 2.1 and 
3.2 and stated that it has procedures already in place to address Recommendation 1.2. Based on 
additional documentation provided to address the discrepancies identified in Finding No. 2, we 
modified the finding reducing the number of discrepancies, accordingly. 2 The reduction in the 
number of discrepancies did not significantly change the finding and, as such, required no change to 
the recommendations. FLDOE' s comments on the draft report are summarized at the end of each 
finding and included in their entirety as Enclosure 2 to this report. 



2 Measurement Incorporated (MI), the handscoring subcontractor to FLDOE' s assessment contractor (CTB), provided 
additional documentation supporting the degree for 9 handscorers from the Nashville regional site; 1 handscorer for the 
Tampa regional site and 9 handscorers for the Durham regional site. In addition, MI provided 8 interview sheets used to 
document the applicant's recommendation for employment - the interviewer is required to document that proof of a degree 
is in the applicant folder. 
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BACKGROUND 



ESEA § 1111(b)(3) requires States to implement a set of yearly academic assessments. The 
assessments are used as the primary means of determining adequate yearly progress (AYP) of the 
State and each of its local educational agencies (LEAs) and schools in enabling all children to meet 
the State's student academic achievement standards. States must use the assessments to measure the 
achievement of students against State academic content and student academic achievement standards 
in Mathematics, Reading or Language Arts, and Science. ESEA § 1 1 1 l(b)(3)(C)(iii) states that these 
assessments shall be used for purposes for which such assessments are valid and reliable, and 
consistent with relevant, nationally recognized professional and technical standards. In June 2007, the 
Department found that Florida's assessment system (not including alternate assessments) met all 
ESEA requirements. 

Section 1111(b)(3) of the ESEA also requires accurate, reliable, high-quality assessment data. 
Assessments are used to hold schools accountable for student achievement. For the 2007 award year/ 
FLDOE received $15.9 million in ESEA Title VI funds for State assessments; and $18.48 million for 
Individuals with Disabilities (IDEA) related activities, of which $306,000 was used for assessment 
testing. 

The Standards for Educational and Psychological Testing 4 (Standards) differentiates between high- 
and low-stakes testing based upon the importance of the results for individuals, organizations, and 
groups. According to the Standards 

At the individual level, when significant educational paths or choices of an individual 
are directly affected by test performance, such as whether a student is promoted or 
retained at a grade level, graduated, or admitted or placed into a desired program, the 
test use is said to have high stakes.. . . Testing programs for institutions can have high 
stakes when aggregate performance of a sample or of the entire population of test takers 
is used to infer the quality of services provided, and decisions are made about 
institutional status, rewards, or sanctions based on the test results .... The higher the 
stakes associated with a given test use, the more important it is that test-based 
inferences are supported with strong evidence of technical quality. 

Accordingly, State assessments required by ESEA are considered high-stakes for States, LEAs, and 
schools for the purposes of calculating and reporting AYP. However, depending on the use of the 
results, these assessments may be considered high-stakes for individual students. 



3 The 2007 award year for Federal funds is July 1, 2007, to September 30, 2008. 

4 The Standards for Educational and Psychological Testing (1999) were developed jointly by the American Educational 
Research Association, the American Psychological Association, and the National Council on Measurement in Education. 
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FLDOE State Assessments 

FLDOE uses the FCAT to assess student achievement in grades 3 through 11. The FCAT consists of 
criterion-referenced tests measuring benchmarks from the Sunshine State Standards 5 (SSS) in four 
content areas - Mathematics (FCAT Mathematics), Reading (FCAT Reading), Science (FCAT 
Science), and Writing (FCAT Writing +). FLDOE administers the FCAT Writing+ assessment in 
February and the FCAT Reading, Mathematics, and Science assessments in March. Students' mastery 
of the content areas is evaluated by multiple choice, gridded-response, extended-response, and essay 
test items. 

FCAT results, which are typically released to school districts by early May, play an instrumental role 
in 1) third grade promotions, 2) deciding whether high school seniors earn a diploma, 6 3) grading 
Florida's public schools, and 4) calculating AYP. As a result, the FCAT is considered high-stakes not 
only for FLDOE, LEAs, and schools, but for individual students as well. 

The FCAT is scored through the coordination of the following three entities 

• CTB McGraw-Hill (CTB) - FLDOE entered into a $131.9 million contract with CTB for the 
period March 31, 2005, to November 30, 2010. Based on the contract, CTB is responsible for 
completing administrative work tasks and activities required for developing, printing, and 
distributing ancillary material; printing, distributing, and retrieving test books and answer 
documents; scanning and scoring answer documents; imaging and handscoring responses to 
performance tasks; and designing, printing, and distributing reports of results in Reading, 
Mathematics, Science, and Writing at selected grade levels of the FCAT. 

n 

• Measurement Incorporated (MI) - CTB entered into a contract with MI for the period 
November 30, 2005, to November 29, 2008. MI is responsible for handscoring FCAT 
Writing+. MI is also responsible for securing test materials, hiring and training readers based 
on approved rubrics and anchor sets, and maintaining an acceptable level of inter-rater 
reliability with scoring personnel and the State. 

• Center for Advancement of Learning and Assessment - Florida State University 
(CALA-FSU) - FLDOE entered into a $1.5 million contract with CALA-FSU effective 
February 1, 2007, to January 31, 2010. CALA-FSU conducts an independent, third party 
review of FCAT results from the scoring contractor (CTB) and subcontractor (MI). 



The SSS were approved by the State Board of Education in 1996 and provide guidelines for Florida's curriculum 
framework. 

6 Grade 10 students are given six opportunities to pass the FCAT. If students have not passed the FCAT by Grade 12, they 
will not receive their high school diplomas, unless their American College Testing (ACT) score is sufficient to waive the 
FCAT requirement or they receive a passing grade on the FCAT retake examination. 

7 The amount of the contract is proprietary information since the contract is between CTB and MI, not FLDOE and MI. 
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FLDOE's contracts for assessment services total approximately $133.4 million. The following Table 
provides a summary of FLDOE contracted services and the associated award amount for assessment 
contractors. 



Table - FLDOE Assessment Contract Expenditures 



Contractor 


Assessment/Service 


Total Amount of 
Contract 8 


Federal 
Expenditures To 
Date 


CTB McGraw-Hill 


FCAT Administration 


$131,916,000 


$52,719,000 


CALA-FSU 


Independent Review 


1,500,000 


y 


Total 


$133,416,000 


$52,719,000 



FLDOE Scoring Process 

The FCAT is scanned at CTB's regional scanning facilities by temporary employees. Apple One, a 
human resource firm, hires seasonal employees to perform CTB's warehouse and scanning operation 
functions. The scanning process captures students' responses for multiple choice and gridded 
response items as well as images of handwritten responses to performance task items. Data pertaining 
to the multiple choice and gridded responses are electronically scored in CTB's mainframe. However, 
written responses are scored by handscorers. 

CTB is responsible for handscoring the Reading, Mathematics, and Science performance tasks and 
subcontracts with MI to score Writing+. Although CTB and MI hire their own scorers, 10 both 
assessment contractors must ensure that all scorers have a bachelor's degree in the content area or 
field related to the subject area being scored; participate in a training program wherein they score 
papers under the supervision of an experienced scoring director and an FLDOE content area expert; 
and pass qualifying tests before being hired. Candidates selected for hire receive other training and 
undergo quality control checks to include supervisory review of their work; pseudo scoring; and, 
when necessary, retraining. FLDOE monitors the inter-rater reliability of scorers through a live, 
secure File Transfer Protocol site. Scorers that do not maintain an acceptable level of scoring 
accuracy are dismissed. 

Several controls are included in FLDOE's scoring process to ensure accurate and reliable reports of 
FCAT results. Specifically, 

• During each FCAT administration, CTB is contractually obligated to develop a data 

verification plan. One component of the plan, mock data processing, tests that all scanning, 
editing, scoring, and reporting functions are working properly prior to live scoring. FLDOE 
performs a number of checks to ensure the accuracy of the answer keys. 



All dollar amounts rounded to the nearest $1,000. 

9 Federal funds are not used to fund the CALA-FSU contract. 

10 Kelly Services provides scoring candidates to CTB by a contractual arrangement. MI is directly responsible for hiring 
its scorers. 
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• Before official scores are released, FLDOE's Data Analysis Reports and Psychometric 
Services Team crosscheck students' individual responses to the answer keys, compare the 
scale scores 11 submitted by CTB to the scale scores computed by FLDOE, and discuss the 
results with CTB. All verification procedures are conducted using a different program and 
software than that used by CTB. 

In addition, FLDOE contracts with CALA-FSU for a third-party, independent test of student scores. 
At the LEA level, school districts are provided an opportunity to verify the accuracy of demographic 
data and review individual student performance. LEAs may also request that the State review a 
student's score before the scores are finalized and published. 



11 The score used to report test results on the entire test. Scale scores on the FCAT Sunshine State Standards tests are 100- 
500 for each grade level and content area. A computer program is used to analyze student responses and to compute the 
scale score. The scale score is used on the Student and Parent Reports for Science and Writing. 
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AUDIT RESULTS 



FLDOE has internal controls over scoring of the FCAT assessment to provide reasonable assurance 
that assessment results are reliable. However, we found discrepancies in the FCAT gridded responses 
and insufficient monitoring of FLDOE' s contractor to ensure compliance with contract requirements. 
In addition, FLDOE' s contractor limited access to documentation required for our audit, delaying the 
audit. 

FINDING NO. 1 - Florida Comprehensive Assessment Test (FCAT) Gridded Response 
Discrepancies 

12 

We identified 9 gridded response discrepancies related to the FCAT Reading and Mathematics test 
and answer booklets for the sample of 50 students reviewed. 13 The gridded response question 14 
requires students to select a numeric answer and shade in the corresponding bubble on a bubbled grid. 
In 6 of the discrepancies, the students' initial response appeared to be erased clearly enough that the 
scanner should have recorded only the darkened revised response, but it did not. In 1 of the 
discrepancies, the scanner picked up a response that was not selected by the student. In 2 of the 
discrepancies, students did not completely erase the first response and 2 bubbles were dark enough for 
the scanner to record both responses, but it did not pick up either of the responses. As a result, one 
sixth grade student did not receive credit for a correct response. 

ESEA § 1 1 1 l(b)(3)(C)(iii) states that assessments shall "be used for purposes for which such 
assessments are valid and reliable, and be consistent with relevant, nationally recognized professional 
and technical standards." 

FLDOE acknowledged that incomplete erasures are problematic. According to FLDOE officials, 
some erasers leave enough carbon residue for the scanner to pick up the erasure, others do not. 
Specifically, the CTB Technology Intensity Resolution Algorithm setting (i.e., the scanner's reading 
setting) should be at a level so the scanner 

• recognizes as a mark any response with an intensity level 15 of 5 or above on the 
15 -point scale used by the scanner, 

• is especially sensitive to whether or not the middle of the bubble is filled in, and 

• will choose between 2 marks in a column if the 2 are 3 or more intensity-levels apart. 



These discrepancies related to test booklets for 8 of the 50 students sampled. 

13 Gridded responses appear on the FCAT SSS Mathematics test in Grades 5-10 and the FCAT SSS Science test in Grades 
8 and 11. We tested Grades 5-10 Mathematics only. 

14 This test item appears in the FCAT Mathematics test in Grades 5-10 and the FCAT Science test in Grades 8 and 11. 

15 The intensity level is a measure of the darkness of the students shading of the bubble in response to the question. 
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For example, in a 5-column, gridded-response item, the student must be careful to completely erase 
when changing an answer, because each column is judged independently by the scanner. The student 
should shade the changed answer as darkly as possible. In addition, when an erasure has no other 
bubble in the column, it will likely be picked up unless it is thoroughly erased. 

As a result of the discrepancies identified, students may not be receiving proper credit for their 
responses based on the intensity, or lack thereof, of an erasure. Inaccurate scanning of gridded 
responses could affect the individual student's overall score and potentially otherwise impact the 
student given the high- stakes implications of the FCAT. 

Recommendations 

We recommend that the Assistant Secretary of the Office of Elementary and Secondary Education 
require FLDOE to 

1.1 Ensure the contractor is correctly setting the Technology Intensity Calibration Algorithm to 
capture students' gridded responses in the scanner. For responses that are manually entered, 
have a second verification of the entry to ensure the gridded response items are captured 
correctly. 

1 .2 Implement procedures to test a sample of the gridded responses during live scoring to ensure 
students' gridded responses are accurately scanned. 

FLDOE Comments 

FLDOE did not concur with Finding 1 . FLDOE did not specify whether or not it concurred with the 
related recommendations but rather indicated that it already had procedures in place to address the 
recommendations . 

In its comments, FLDOE stated that it employs an answer document decoding process that uses 
nationally recognized professional and technical standards and state-of-the-art scanning technology to 
ensure that assessment results are reported accurately. FLDOE' s comments to the draft report 
included the following 4 points related to the finding. 

1. FLDOE took exception to our methodology for validating the scanning accuracy of gridded 
responses using copies of answer documents rather than reviewing and/or 

re-scanning original documents. Specifically, FLDOE asserted that the pages of the test 
booklets do not photocopy well and it would be difficult for the human eye to discern the 
difference between an erasure and a stray particle. 

2. FLDOE indicated that current practice is not fully or accurately described in the audit report 
related to this finding. Specifically, the scanner has built in checks for miscalibration; the 
assessment contractor CTB follows standard operating procedures for scanner calibration 
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including recalibrations after every 5,000 scans; FLDOE staff are present at the scoring site to 
perform an early scan check to ensure scanning accuracy; and student demographic data and 
response arrays on original answer documents are compared to the electronic scan file to 
ensure that documents are accurately scanned. 

3. FLDOE has 2 processes for resolving scanning errors missed during front-end checks. First, 
an automatic manual rescore of tests for Grade 12 - Adult Retake students who have failed by 
25 or fewer points; and second, the districts are allowed to submit a request to have student 
scores reviewed for scoring anomalies or scanning problems. 

4. Prior to the test administrations, students are instructed on responding to multiple choice items, 
erasing completely, completing gridded response items, and checking their marks. 

Although acknowledging potential quality control issues with key entry data, FLDOE stated that its 
scoring specifications and the contractor's procedures require subsequent verification of the data 
manually entered into the system. 

FLDOE stated that the new assessment contractor, Pearson, will be required to ensure the calibration 
of the scanner is completed and documented to show the frequency of the scanner calibrations. 

OIG Comments 

FLDOE provided no additional information that would require a change to Finding No. 1 and the 
related recommendations. The methodology of reviewing copies of test booklets was employed to 
respond to limitations placed on the audit by the assessment contractor, CTB. Basically, the audit 
team had 4 days to pull a sample of 84 FCAT test and answer booklets, which could not be removed 
from the contractor's warehouse. Copies of the sample documents were required to allow the auditors 
to adequately review the sample. However, we provided copies of all sample documents with 
discrepancies to FLDOE and adjusted our finding when FLDOE provided an original document that 
resolved the discrepancy. Although FLDOE had the opportunity to resolve any discrepancy with 
original documentation, it only resolved 1 discrepancy - the remaining discrepancies noted in the 
finding were not resolved. 

Our report acknowledges that FLDOE has an automatic manual rescore process for Grades 12- Adult 
Retake students who have failed the FCAT Reading and Mathematics test by 25 or fewer scale score 
points. However, this automatic manual rescore process does not include grades 3 through 10. Since 
the FCAT is considered a high-stakes exam and affects promotion for grades 3 and 10, it is important 
that FLDOE implement procedures to capture all student responses correctly. At a minimum, FLDOE 
should expand its manual rescore process to include grades where the FCAT is considered a high- 
stakes examination. 
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Although standard operating procedures for scanner calibration include recalibrations after every 
5,000 scans, FLDOE did not include in its response the scanning and calibration reports to validate 
that the assessment contractor followed those procedures. In addition, as reported in the Other 
Matters section of this report, FLDOE does not consider the "specifications," which detail the 
calibration requirements, as part of the assessment contractor's contractual requirements. 

FLDOE addressed Recommendation 1.1 by including requirements in its contract with the new 
assessment contractor, Pearson. Specifically, FLDOE stated it will require the contractor to perform 
scanning calibrations and document the procedures to show the frequency of the scanner calibrations. 
However, FLDOE did not include in its response a copy of the new contract for us to validate that the 
requirement is included. 

FLDOE indicated that it has procedures in place to address Recommendation 1.2. Specifically, 
FLDOE stated that it tests a sample of the gridded responses prior to each test administration to ensure 
students' gridded responses are accurately scanned. We observed the testing while on-site prior to the 
administration of the test. However, despite the testing done by FLDOE, we identified a number of 
gridded response discrepancies in this report. As such, FLDOE should review its methodology for 
sampling gridded responses and perform the testing during live scoring. 

FINDING NO. 2 - Insufficient Monitoring of Florida Department of Education's 

(FLDOE's) Contractor 

FLDOE did not sufficiently monitor its assessment contractor, CTB. Specifically, we found that 
FLDOE was not aware that CTB (1) had inadequate safeguards for discarding FCAT Student PII; 
(2) did not comply with contract terms; and (3) did not have adequate document control procedures. 

Inadequate Safeguards for Discarding FCAT Student PII 

During our visit to the CTB contractor facility in Indianapolis, Indiana, we found that student PII was 
plainly visible after being discarded in the trash. Specifically, CTB had inadequate controls for 
safeguarding PII, an issue that we identified in two of the three States selected for review. CTB 
officials acknowledged the incident and immediately issued a Quality Alert Memorandum and took 
steps to resolve the issue. However, the contractor needs to implement additional procedures, 
including secure trash pickup or shredders at each quality control workstation to better safeguard 
FCAT material. 

According to the Forum Guide to Protecting the Privacy of Student Information: State and Local 
Education Agencies, issued by the National Forum on Education Statistics, agency and school 
personnel are legally and ethically obligated to safeguard the confidentiality of student data. In 
addition, a policy letter issued by the Director of the Federal Policy Compliance Office in 
January 2008 states that when contractors are used to provide services that require access to student 
PII, they are bound by the same restrictions on disclosure and destruction of information that apply to 
the SEA. The SEA is also responsible for ensuring that its contractors do not disclose PII or allow 
any other party to have access to any PII from education records. Although the FLDOE contract 
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included a statement on the Family Educational Rights and Privacy Act (FERPA), CTB did not 
comply with the contract and, as a result, we found PII documentation discarded and plainly visible in 
the contractor's trash. FLDOE's current practice for printing or displaying student social security 
numbers instead of other unique identifiers on assessment documents demands a higher degree of 
diligence from everyone handling those documents. 

Non-compliance with Contract Terms 

Although required by contract, CTB did not 1) notify FLDOE of changes to subcontractors; 2) have a 
disaster recovery plan available at the Indianapolis facility; and 3) sufficiently monitor its 
subcontractors. As a result, CTB was not in compliance with contract terms. 

The FLDOE contract requires CTB to 1) receive advance, written consent from FLDOE prior to 
assigning or subcontracting all or any portion of the contract; 2) have a disaster recovery plan on hand 
at the facility; and 3) monitor subcontracts. In addition, 34 C.F.R. § 80.40 states 

Grantees are responsible for managing the day-to-day operations of grant and 
subgrant supported activities. Grantees must monitor grant and subgrant supported 
activities to assure compliance with applicable Federal requirements and that 
performance goals are being achieved. Grantee monitoring must cover each program, 
function or activity. 

Change of Subcontractor Without Notification. The CTB Temporary Employment Services 
Agreement specified Kelly Services as the subcontractor responsible for hiring temporary personnel 
for scanning/scoring operations with CTB. However, CTB changed from Kelly Services to 
AppleOne 16 without notifying FLDOE. The FLDOE assessment contract states that "the Contractor 
may not assign or subcontract all or any portion of this Contract without the advance written consent 
of the Department." However, FLDOE officials were not notified that CTB changed its subcontractor 
for hiring operations personnel from Kelly Services to AppleOne. 

No Disaster Recovery Plan . The Indianapolis, Indiana, scoring operations facility does not have a 
disaster recovery plan. According to the FLDOE contract with CTB, a disaster recovery plan was 
required to be on hand at the facility. In addition, 34 C.F.R. § 80.36 (b)(2) states 

Grantees and subgrantees will maintain a contract administration system which ensures 
that contractors perform in accordance with the terms, conditions, and specifications of 
their contracts or purchase orders. 

Although the General Manager described procedures and stated that the disaster recovery plan 
was maintained at the CTB office in Monterrey, California, we did not receive a copy of the 
plan. 



Howroyd-Wright Employment Agency, Inc. dba AppleOne Employment Services (AppleOne). AppleOne services 
include temporary, direct hire, and temp-to-hire staffing; employee screening and training; payroll and tax filing; and time 
and attendance processing. AppleOne holds a contract with the U.S. General Services Administration (Contract Number: 
GS-02F-001 1U) effective October 2007 and due to expire in October 2012. 
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Insufficient Monitoring of Subcontractors. CTB did not sufficiently monitor its subcontractors, 
AppleOne and MI, to ensure that they hired qualified employees. As a result, we found that 14 
AppleOne employees were not qualified to be hired and 17 MI readers should not have been allowed 
to score the Writing+ test. 

CTB's General Manager stated that the subcontract requires that operational personnel hired to 
process the FCAT receive a qualifying score on AppleOne' s basic skills test (Prove It). However, 
CTB did not sufficiently monitor AppleOne to ensure that its employees passed the test, and we found 
that 14 of 72 employees did not pass the test. 

Although CTB is responsible for the Writing+ test that is scored by the subcontractor, MI, CTB did 
not sufficiently monitor MI to ensure that its employees had the required degrees to score the FCAT 
Writing + test. The FLDOE RFP and Contract Specification require that "All scorers will have at 
least a bachelor's degree in the FCAT content area or related fields of English, Mathematics, Science 
or Education to perform testing." In addition, in 2007, FLDOE amended the contract, requiring CTB, 
and in turn, MI to hire a third party independent contractor to verify the degree information of 
handscoring applicants. We reviewed 647 MI training files and found that MI was unable to provide 
adequate documentation showing that its handscorers obtained bachelor' s degrees or that the degree 
information was verified in a timely manner. Of the 647 reviewed, we found that 16 MI readers 17 
should not have been allowed to score the Writing+ test. Specifically, 

• Of the 206 handscorers from the Nashville 18 regional site, 15 did not have degree verification 
by a third party independent contractor completed until 2009; 4 did not have a copy of a 
degree or transcript 19 in the file. 

• Of the 241 handscorers from the Tampa 20 regional site, 1 did not have degree verification by a 
third party independent contractor completed until 2009 and 3 did not have a copy of the 
degree or transcript in the file. 

• All 200 handscorers from the Durham regional site had their verifications by a third party 
independent contractor completed in 2007 or 2008 (prior to the FCAT scoring). However, 2 
did not have a copy of the degree or transcript in the file and 3 did not have a copy of the MI 
interview sheet 22 in the file. 



As indicated in the bullets, the 16 consists of the 15 MI employees from the Nashville field office and 1 MI employee 
from the Tampa field office, who did not complete verifications until after scoring had been completed. 

18 The Nashville Regional office tests the fourth grade writing. 

19 We accepted the transcript and teaching certificate if it stated Bachelor's of Science or Bachelor's of Arts degree 
conferred with the year indicated. 

20 The Tampa Regional office tests the eighth grade writing. 

21 The Durham Regional office tests the tenth grade writing. 

22 The MI Interview sheet is used to document the applicant's recommendation for employment. On the interview sheet is 
a documentation review/applicant briefing box. The interviewer is required to document that proof of a degree is in the 
applicant folder. 
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Although FLDOE tests a 20 percent sample of FCAT Writing+ handscorers during the testing season, 
it does not verify names to transcripts. MI through CTB provides FLDOE an employee number as the 
identifier and copies of transcript information on an employee sheet. FLDOE does not see or verify 
names or transcripts to ensure that personnel are qualified to score the FCAT Writing+ exam. FLDOE 
officials stated that because of Florida's open records laws, the MI/CTB employee handscorers degree 
information is PII which FLDOE does not want to be responsible for maintaining. According to State 
officials, FLDOE relies on its assessment contractors to verify the degree requirements and ensure that 
personnel are qualified to perform the scoring. 

Inadequate Document Control Procedures 

CTB does not have adequate document control procedures at the Indianapolis, Indiana, facility. We 
requested that FLDOE provide the FCAT Reading and Mathematics test and answer books for 50 
students. 23 However, FLDOE was not able to convince its assessment contractor to provide the 
requested information, and the assessment contractor, CTB, was not willing to retrieve the answer 
books without payment. After CTB refused to provide required documentation necessary to address 
the objective of our audit, we ultimately retrieved the data without contractor assistance. However, 
the retrieval of the required documentation was hindered by unreliable information provided through 
CTB's document control system (ASAP 24 ). 

FLDOE officials stated the FCAT booklets were archived in the CTB warehouse and retrieval of the 
test booklets for a Federal audit was not part of the FLDOE contract with the CTB contractor. CTB 
provided a worksheet request for payment of $3,750 to retrieve the FCAT sample. Since CTB was 
unwilling to provide the requested data without payment, we issued an OIG subpoena to obtain the 
information needed for our audit. 

In response to our January 2009 subpoena for the required documentation, CTB again requested 
payment of $3,750 to retrieve the sample. In subsequent discussions, CTB agreed to provide access to 
the information if audit staff would retrieve the test booklets themselves from its warehouse. 
However, CTB's document control system was unreliable and contained incorrect information. 

After receiving 4 separate reports from CTB related to the location of the FCAT booklets in the cages 
and CTB's document control personnel pulling the wrong cages, we were able to retrieve only 70 of 
the selected 84 answer booklets 25 while on-site. In addition, we noted the following issues with 
CTB's document control system - (1) rows were not numbered in 7 cages; (2) the reports provided by 
CTB included 5 Science books even though Science books were not included in the sample; (3) CTB 



Our sample of Reading and Mathematics tests for 50 students would have totaled 100 single test and answer booklets 
(50 Reading plus 50 Mathematics) but instead totaled 84 single test and answer booklets because the sixth, seventh, and 
ninth grade booklets were one combined Reading and Mathematics test. 

24 ASAP is the name of CTB's document retrieval system and is not an acronym. 

25 Our sample of 50 Reading and Mathematics tests equated to 84 single test and answer booklets because the sixth, 
seventh, and ninth grade booklets were one combined Reading and Mathematics test. 
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had to "re-cage" a cage to find 1 answer booklet; and (4) we located 1 book in a row different than 
the row reported by the ASAP system. 

We requested CTB provide the remaining sample of 14 FCAT answer booklets that we were unable to 
locate. Of the 14 FCAT booklets CTB retrieved, we noted that there were problems with 2 of the test 
booklets. One test booklet had 2 pages missing, and the wrong FCAT booklet was provided in the 
other instance, requiring a third request to FLDOE and CTB to receive the complete test booklets. 

FLDOE did not sufficiently monitor the assessment contract to ensure that CTB 1) safeguarded FCAT 
student PII information; 2) complied with contract terms, including monitoring subcontractors; and 
3) had adequate document control procedures to locate assessment documentation from the storage 
facility. FLDOE officials stated that the main concern for the State is that assessment scores reported 
are reliable and accurate. FLDOE acknowledged the issues the audit team found but stated that the 
issues identified did not affect the accuracy of the reporting of FCAT scores. 

FLDOE included penalties for non-compliance in the contract for critical activities and extremely 
critical work tasks; however, there are no penalties in the contract for the non-compliance identified in 
this finding. Because of the high-stakes nature of the FCAT assessment program, all aspects of test 
administration, scoring, and reporting require proper diligence toward maintaining test security, as 
well as the accuracy and reliability of test results. FLDOE should document and monitor contractor 
performance in areas that are not currently reviewed yet are important to reliability of the assessment 
results. 

Recommendations 

We recommend that the Assistant Secretary of the Office of Elementary and Secondary Education 
require FLDOE to 

2.1 Use unique identifiers instead of student social security numbers and dates of birth on 
assessment documents. 

2.2 Ensure that all contractors are aware of the proper handling of PII and include language in their 
contracts to properly address the correct handling procedures related to the disposal of student 
PII. 

2.3 Monitor the contractor to ensure compliance with contract provisions and include a table of 
penalties in the contract for non-compliance with contractual requirements. 

2.4 Monitor document control procedures at the contractor facilities at least annually. 



Document control personnel had to remove every FCAT booklet from a cage, rescan, renumber, and place back in the 
proper order to find the one FCAT answer booklet. 
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FLDOE Comments 

FLDOE generally concurred with Finding No. 2. Although FLDOE' s comments did not specifically 
address whether it concurred with the recommendations, the comments indicated that FLDOE had 
implemented actions to address Recommendations 2.2, 2.3, and 2.4, but did not implement any action 
at this time to address Recommendation 2.1. 

FLDOE stated that it enforces the protection of PII and requires its contractors to protect PII as well. 
FLDOE disagreed that our finding PII discarded in a non-secure fashion merited the conclusion that, 
"FLDOE did not sufficiently monitor its assessment contractor." FLDOE stated that its Scoring and 
Reporting personnel are present at scanning sites during the scoring process of each FCAT 
administration to ensure that all aspects of the scanning operation proceed according to its contractual 
agreement and that related requirements are met. FLDOE stated that it regretted that the cited incident 
occurred but did not think the one instance warranted the audit team's conclusion. 

FLDOE disagreed that CTB's change from Kelly Services to Apple One for hiring operations violated 
contractual requirements. Although FLDOE' s comments included the contract section requiring any 
change of subcontractors to be approved in advance, FLDOE stated that neither Kelly Services nor 
Apple One were included in CTB's chart of subcontractors and services. According to FLDOE, CTB 
explained that it had long-standing relationships with both agencies for hiring temporary employees 
and, as such, were not required to list them on the proposal. FLDOE agreed that it is not unreasonable 
that CTB only request approval for changes to subcontractors identified in the proposal. 

FLDOE concurred that the contractor should have been able to provide a copy of the Disaster 
Recovery Plan and added that it has requested written assurance from the current assessment 
contractor, Pearson, that the Plan is readily available for review at all scanning sites. FLDOE stated 
that verification of the Disaster Recovery Plan will be added to its monitoring checklist. 

FLDOE stated that it reviewed a random sample of 20 percent of scorer candidates' qualifications and 
either approved or dismissed candidates based on the results of the review. FLDOE acknowledged 
that a review of the entire pool of candidates may have resulted in findings similar to the audit 
findings, but the sample of 20 percent was agreed to by FLDOE and CTB in recognition of the time 
critical factor and human resources required to conduct the reviews. 

According to FLDOE, the assessment contractor complied with all document control procedures 
called for in the contract. While it acknowledged that the audit team's experience indicated some 
corporate inefficiencies, FLDOE maintained that its needs for document retrieval have been met. 

Regarding Recommendation 2.1, FLDOE stated that it would continue to use the process currently in 
place to ensure timely and accurate reporting of student results and maintain stringent security policies 
and procedures, but would explore the potential use of alternative identifiers for students. 
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Regarding Recommendation 2.2, FLDOE included an excerpt from the contract with the new 
assessment contractor, Pearson, which included requirements for properly handling PH. 

Regarding Recommendation 2.3, FLDOE stated that it has several quality assurance measures 
including performance metrics to hold the contractor accountable for contractual obligations. FLDOE 
added that if requirements are not met, payment is withheld; and it provided an excerpt from its 
current contract with Pearson documenting that provision in the contract. 

Regarding Recommendation 2.4, FLDOE stated that is has added specific items and processes to its 
monitoring checklist, including verification of procedures used by the contractor to manage and 
dispose of PII and verification of the Disaster Recovery Plan. 

OIG Comments 

Based on additional documentation that the handscoring subcontractor provided separately from 
FLDOE' s comments, we modified the finding. Specifically, MI, the handscoring subcontractor of 
FLDOE' s assessment contractor (CTB), provided additional documentation supporting the degree for 
9 handscorers from the Nashville regional site; 1 handscorer for the Tampa regional site and 9 
handscorers for the Durham regional site. In addition, MI provided 8 interview sheets used to 
document the applicant's recommendation for employment - the interviewer is required to document 
that proof of a degree is in the applicant folder. Based on the additional documentation, we reduced 
the number of discrepancies, accordingly. The reduction in the number of discrepancies did not 
significantly change the finding and, as such, required no change to the recommendations. 

In response to FLDOE' s contract requirements for properly handling PII in its current contract with 
Pearson, we noted the same language in its contract with CTB and still found student PII information 
discarded in a trash receptacle. Because FLDOE stated that it will continue to use PII information 
while exploring alternative identifiers for students, it is important that FLDOE provide oversight and 
monitoring to ensure compliance with the contract's requirements for handling PII. 

Although FLDOE can waive contract requirements, we maintain that CTB's change from Kelly 
Services to Apple One for hiring operations violated explicit provisions in the contract. The contract 
did not exclude long-standing contract relationships from the required approval and, therefore, CTB 
should have requested FLDOE' s approval to change subcontractors. 

FLDOE stated that it reviewed a random sample of 20 percent of scorer candidates' qualifications and 
either approved or dismissed candidates based on the results of the review. However, based on the 
number of discrepancies identified in our audit, FLDOE should consider re-evaluating its scorer 
qualification review methodology. 

In response to FLDOE' s contract requirements for quality assurance measures, including performance 
metrics to hold the contractor accountable for contractual obligations and withholding payment when 
requirements are not met, we noted that the same language is in its contract with CTB. However, the 
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liquidated damages section of the contractor did not address any of the discrepancies noted in our 
audit report. Therefore, it is important that FLDOE monitor the contractor to ensure compliance with 
the contract. FLDOE should consider adding language related to standard penalties for general non- 
compliance so that in encompasses all contractual requirements. 

Although FLDOE indicated that it has added specific items and processes to its monitoring checklist, 
it did not include in its response a copy of the revised monitoring checklist for our review. 

FINDING NO. 3 - FLDOE'S Contractor Delayed Federal Audit by Limiting Access to 
Assessment Documentation 

FLDOE and its assessment contractor did not provide access to documentation required to address the 
objectives of our Federal audit. In our effort to obtain documentation from the contractor, we found 
that FLDOE' s contracts were not adequate to facilitate a Federal audit. FLDOE had a contract with 
CTB for the FCAT Reading, Math, Science, and Writing+ assessments. However, the contract did not 
include the required audit clause 27 allowing access to data for audits by Federal officials despite the 
fact that FLDOE contracted with these vendors to assist in its effort to meet Federal requirements 
under ESEA § 1111, and each contract was paid, at least in part, using more than $52.7 million in 
Federal funds (as shown previously in the Table). 

In addition to test booklets, we requested that the assessment contractor and subcontractors provide all 
training files, degrees, and qualifying score sets for contractor personnel that scored the Spring 2008 
FCAT. The assessment contractor and subcontractors stated that the information was proprietary and 
could not be provided to the audit team and that the retrieval of the test booklets for our audit was not 
part of the FLDOE contract and would require an additional payment. Even though the contracts for 
assessment services with CTB and Harcourt contained clauses authorizing State audits, FLDOE was 
unable to obtain this documentation from its contractors for use in our Federal audit. 

As a result, we issued subpoenas to CTB and two of its subcontractors to provide the required source 
documentation. In response to the subpoenas, the two subcontractors complied and provided the 
required source documentation; and CTB, the assessment contractor, complied with a portion of 

the subpoena. However, CTB provided a work statement and price quotation for $3,750 to retrieve 
the audit sample requested to complete our audit. Although CTB was not willing to provide the 
required information without payment, it allowed the audit team to retrieve the audit sample 
themselves. 

The Inspector General Act of 1978 states - 

In addition to the authority otherwise provided by this Act, each Inspector General, in 
carrying out the provisions of this Act, is authorized — 



Required by 34 C.F.R. § 80.36 (i)(10). 



Final Report 
ED-OIG/A04I0043 



Page 18 of 36 



(1) to have access to all records, reports, audits, reviews, documents, papers, 
recommendations, or other material available to the applicable establishment which 
relate to programs and operations with respect to which that Inspector General has 
responsibilities under this Act. 

34 C.F.R. § 80.36 (i)( 10) provides that a grantee's contracts must contain a provision requiring access 
by the grantee, the Federal grantor agency, or any duly authorized representatives to any books, 
documents, papers, and records of the contractor which are directly pertinent to that specific contract 
for the purpose of making audit." 

Assessment contractors play a large role in the scoring of FLDOE's State assessments. Because 
FLDOE did not include the audit clause allowing access to documentation for Federal audits in its 
contracts with assessment vendors and the vendors would not comply with our request for 
information, we issued subpoenas to FLDOE's primary contractors and two subcontractors to compel 
them to provide the documentation needed to complete the audit. The need to use OIG subpoena 
authority delayed the audit. If Federal auditors are unable to gain access to records and personnel to 
obtain relevant information while conducting audits, Federal oversight is diminished and Federal 
funds and programs may be at risk. 

Recommendations 

We recommend that the Assistant Secretary of the Office of Elementary and Secondary Education 
require FLDOE to 

3.1 Include a Federal audit clause provision in contracts for Department funded programs. 

3.2 Include a table of penalties in the contract for non-compliance with a Federal audit. 

FLDOE Comments 

FLDOE agreed that it did not include a Federal audit clause in its contract but did not agree that the 
assessment contractor delayed the Federal audit by limiting access to assessment documentation. 
FLDOE agreed to Recommendation 3.1 and stated that it would review its liquidated damages 
section of the contract and determine whether penalties for non-compliance should be included in 
future contracts. 

In its response, FLDOE stated that it and its assessment contractor did not limit access to 
documentation required to address the objectives of the Federal audit. According to FLDOE, from the 
initial site visit by the auditing team to the conclusion of the audit, all requested information and data 
regarding Statewide assessments in Florida were provided. FLDOE stated that, although it suggested 
multiple strategies to physically check a representative sample of original answer documents against 
the electronic scan file during live scanning, the audit team requested the 2- year-old documents from a 
secure warehouse where processed documents are stored and archived. FLDOE added that due to the 
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nature of the request, the contractor may have taken longer to process this request than desired by the 
audit team, but at no point was access denied or limited. 

OIG Comments 

We acknowledge that eventually we were provided access to the information needed to address our 
audit objectives. However, we were forced to issue federal administrative subpoenas for records that 
FLDOE's contractors were required to provide to us by regulation. Further, the access eventually 
granted was limited and under the direct supervision of assessment contractor staff. 

During the course of the audit, we requested test booklets and answer sheets to assess the adequacy of 
controls to ensure accurate scanning and that the scanned data maintained its integrity during export to 
the contractor's system. The assessment contractor and subcontractors initially stated that the 
information requested was proprietary and could not be provided to the audit team. Subsequently, the 
assessment contractor stated that the retrieval of the test booklets for our audit was not part of the 
FLDOE contract and would require an additional payment. In response to subpoenas, the contractor 
allowed us supervised access to the documentation needed to complete our objectives, and the 
subcontractors provided limited information to access qualifications. The extended process used to 
eventually gain access to documentation delayed the audit by several months. 

FLDOE provided a copy of the contract amendment that includes language that addresses the Federal 
audit clause. The contract amendment sufficiently addresses Recommendation 3.1. In addition, 
FLDOE stated it would consider liquidated damages in future contracts in response to 
Recommendation 3.2. FLDOE's assessment contractor limited access to the documentation required 
and delayed our audit while receiving more than $52.7 million in Federal Funds. As such, we 
maintain that FLDOE needs to modify its contract with its new assessment contractor, Pearson, and 
include a liquated damages clause for non-compliance with a Federal audit. 
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OTHER MATTERS 



Error Identified in FLDOE's 2006 Third Grade Reading Test 

In 2007, FLDOE discovered a problem with the 2006 third grade Reading test. FLDOE concluded 
that the problem was a one-time mistake caused by the accidental misplacement of "anchor" questions 
that ensure the FCAT difficulty level is consistent from year to year. In response to the problem, 
FLDOE named a technical advisory committee of district officials and testing experts, who in turn 
hired the Buros Center for Testing at the University of Nebraska to conduct an independent review of 
the issue. In a report released in December 2007, the Center stated that the error was caused by an 
"anchor" issue, which had been corrected. Although FLDOE initiated procedures to identify and 
correct the issue from 2006, the incident was published in Florida newspapers, and user confidence in 
the reliability of FCAT results may have been undermined. The accuracy and reliability of assessment 
data are important because if users do not perceive FCAT results as reliable, they may not use the 
assessments to identify the specific academic needs of individual students, as intended by Federal law. 
In addition, we found discrepancies during our audit, which confirm that FLDOE needs to implement 
additional controls to ensure contractors are adhering to contract requirements. 

FCAT Specifications Not Part of Contractual Obligations 

During the course of our audit, we learned that FLDOE does not consider the FCAT specifications 28 
in the contract as a contractual requirement. Specifically, FLDOE officials stated 

• The specifications are contained in a living document that changes frequently. 

• FLDOE does not have the manpower or time to change the specifications every time a change 
is found to be necessary. 

• There are critical tasks in the specifications that are contractual and liabilities will be assessed 
for non-compliance by the contractor. 

FLDOE pays its assessment contractor $131.9 million dollars to administer the FCAT contract. It 
specifically states in the contract and Request for Proposal that all specifications are part of the 
contract terms. Accepting specifications as a contractual requirement is especially important since 
FLDOE has just awarded a new contract for $254 million dollars to Pearson beginning in the 
2009-2010 assessment year. All aspects of the contract should be adhered to according to the terms of 
the contract. 



The FCAT Specifications are the documents that explain in detail the policies and procedures the contractor will use 
during the assessment scoring period. FLDOE provided seven FCAT specifications for 2008 - Handscoring, Data 
Verification, Packing and Distribution, Scoring, Technical Scoring, Reporting, and Pre-ID. 
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OBJECTIVE, SCOPE, AND METHODOLOGY 



The objective of our audit was to determine whether controls over scoring of assessments at the 
FLDOE were adequate to provide reasonable assurance that assessment results were reliable. 
Our review covered assessments administered in school year 2007-2008 that were used for evaluating 
individual student achievement and making Adequate Yearly Progress (AYP) determinations under 
Section 1 1 1 1(b)(3) of the ESEA. 

To accomplish the objective of our audit of FLDOE, we 

• Compiled and reviewed media reports, assessment articles, prior OIG audit reports, prior audit 
reports issued by the Government Accountability Office and the Florida State Comptroller, 
and the monitoring reports and peer review determination letter issued by the Office of 
Elementary and Secondary Education (OESE). 

• Identified Title VI and Title I, Part A funding and contacted the OESE for additional 
information on FLDOE. 

• Reviewed the database of errors compiled by the National Board of Educational Testing and 
Public Policy to identify potential risk areas in FLDOE' s system for scoring State assessments. 

• Reviewed laws, regulations, and guidance applicable to State assessment scoring and student 
privacy rights. 

• Obtained an understanding of the assessments administered by FLDOE, as well as the control 
system in place to ensure reliable student scores through interviews of relevant personnel. 

• Observed Retake scanning at FLDOE' s contractor scanning scoring facility in Indianapolis, 
Indiana. 

• Conducted interviews with personnel from FLDOE and CTB and MI to better understand 
internal control processes and procedures over assessment scoring. 

• Reviewed requests for proposals, vendor proposals, and contracts for the FCAT, as well as 
documentation supporting FLDOE' s procurement process in relation to the contract. 

• Reviewed documentation over the background and administration of the FCAT. 

• Obtained and reviewed FCAT documentation describing the specifications used in processing 
and reporting assessment results. 

• Reviewed the FCAT Pre-ID process that allows FLDOE to provide pre-labeled booklets for 
students taking paper exams and individual user accounts for online testing. 

• Reviewed a list of incident reports at FLDOE school districts regarding FCAT data. 

• Interviewed FLDOE key staff related to ESEA assessments (Reading, Math, Science, and 
Writing+). 

• Documented FLDOE's policies, procedures, and controls over the FCAT scoring process. 

• Interviewed contractor personnel at CTB's Indianapolis, Indiana, scoring facility. 

• Reviewed FCAT request for proposals, contractor proposals, evaluations, and contracts. 
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• Observed and documented controls and security procedures at CALA-FSU and the FLDOE 
Test Control Facility. 

• Observed scanning operations at the CTB central scanning facility, shipping warehouse, and 
cutting facility in Indianapolis, Indiana. 

• Documented policies and procedures related to CTB's scanning procedures. 

• Obtained a copy of the raw data file from FLDOE with the universe of 1.8 million FCAT test 
results (Reading, Math, Writing, and Science) for use in selecting a sample. 

• Obtained a copy of the electronic test define (answer key). 

• Conducted a partial review of the CTB assessment contractors training records. 

• Reviewed documentation provided to support monitoring of contractor services. 

• Issued subpoenas to CTB, MI, and Kelly Services for written policies and procedures, reader 
qualifications documentation, and training records. 

• Reviewed inter-rater reliability for compliance with quality assurance requirements established 
in the contract and Scoring Specifications between FLDOE and CTB. 

• Reviewed qualification documentation provided by MI for the 647 readers who participated in 
training to score the Spring 2008 FCAT Writing assessment. From the documentation 
provided, we verified the degrees obtained from non-foreign educational institutions. 

We selected a random sample of 50 Reading and Mathematics FCAT scored answer records from a 
population of 1,545,696 scored Reading and Mathematics FCAT records. Our combined sample of 50 
Reading and Mathematics tests should have totaled 100 single test and answer booklets (50 Reading 
plus 50 Mathematics) but instead totaled 84 single test and answer booklets because the sixth, seventh, 
and ninth grade booklets were one combined Reading and Mathematics test. We compared the 
hardcopy student answer documents with the electronic data file records provided by FLDOE to 
ensure accurate scanning and that the scanned data maintained its integrity during export to the 
contractor's system. We identified 9 discrepancies related to the 50 FCAT Reading and Mathematics 
scored records as detailed in Finding No. 1. 

We did not use information from a database in order to address our objectives. We used data supplied 
by FLDOE, CTB, MI, and Kelly Services in Microsoft Excel spreadsheets to conduct our testing. 

We reviewed the sufficiency of FLDOE' s contractual safeguards for protecting student information 
during the assessment process in accordance with the Family Educational Rights and Privacy Act 
(FERPA). We found the relevant FERPA student information statements were contained in the 
FLDOE contracts. 

We began our audit work on-site at FLDOE in Tallahassee, Florida, on June 23, 2008. We held an 
exit conference with FLDOE officials on June 4, 2009. Our audit was performed in accordance with 
generally accepted government auditing standards appropriate to the scope of the review described 
above. These standards require that we plan and perform the audit to obtain sufficient, appropriate 
evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. 
We believe that the evidence obtained provides a reasonable basis for our findings and conclusions 
based on our audit objectives. 
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Enclosure 1: Glossary 

Anchor Sets - According to FLDOE's contract between CTB, subcontracted to MI, "sample essays 
which clearly exhibit the criteria for each score point." 

Blind Scoring - According to FLDOE's contract with MI, "a method of scoring that allows each 
student response to be scored independently, without the reader having knowledge of any previous 
scores." 

Equating - According to the Standards for Educational and Psychological Testing, it places two or 
more essentially parallel tests on a common scale. 

Gridded Response - Questions that require students to solve problems and "bubble" or shade their 
numerical answers in answer grids. Answers may be gridded using several correct formats. Students 
must accurately fill in the bubbles below the grids to receive credit for their answers. 

Inter-rater Reliability - According to the Standards for Educational and Psychological Testing, the 
consistency with which two or more judges rate the work or performance of test takers. 

Psychometrician - According to Margin of Error: The Education Testing Industry in the No Child 
Left Era, a report published by Education Sector, experts trained in measurement theory and statistics 
who analyze how test items perform in field trials and conduct other statistical analyses in test- 
making. 

Scale Score - The score used to report student results for the entire test in FCAT Reading, 
Mathematics, and Science. Scale scores on the FCAT range from 100 to 500 at each grade level. 
The scale score is the result of Item Response Theory processing and equating. 

Reliability - According to the Standards for Educational and Psychological Testing, the degree to 
which test scores for a group of test takers are consistent over repeated applications of a measurement 
procedure and hence are inferred to be dependable and repeatable. 

Rubrics - FLDOE's contract with CTB (MI), "scales (currently six-point) used to determine the 
range of excellence in the papers written in response to a given prompt." 
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Enclosure 2: FLDOE Response 



Florida Department of Education 
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Regisnal Inspector General for Aud t 

U.S- Department of Education 

OIG - Office of Audit Services 

61 Forsyth Street, SW. Suite 18T71 

Atanta. Georgie 3D303 

Dear Ma. Wempe: 

In connection with tne stuii; (Control Number ED-OIG/AD4I0043) of tfie Florida Department of 
Education Controls Over State Assessment Scoring by Hie U.S, Department of Education, pleasu 
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FLDOE Response to USED FINDING NO. 1 - Florida Comprehensive Assessment Test (FCAT) 

Gridded-Response Discrepancies 

We disagree with this finding. FLDOE employs an answer document decoding process that uses 
nationally recognized professional and technical standards and state-of-the-art scanning technology to 
ensure that assessment results are reported accurately. Four points related to this finding need to be 
clearly articulated. 

1 . It is our opinion that the methodology used by the auditing team to validate the scanning 
accuracy of gridded responses on FCAT answer documents was not in-depth enough to 
arrive at the conclusions cited in Finding No. 1. Specifically, the auditors determined that 
scanning discrepancies existed by viewing copies of answer documents rather than 
reviewing and/or re-scanning original documents. Copies are susceptible to varying 
degrees of shading, and the human eye cannot consistently and accurately discriminate 
bubble intensity to the fine degree that today's scanning technology can. In addition, to 
ensure that scanners decode student responses instead of printed information, scannable 
books are printed in light shades of ink, usually gray. Due to this, these pages do not 
photocopy well and it would be difficult for the human eye to discern an erasure from a 
stray particle. 

2. Current practice is not fully or accurately described in the audit report as it relates to this 
finding. Specifically, the Scoring Operations Specifications for Spring 2008 FCAT (see 
Exhibit 1) clearly detail that the scanners used by the scoring contractor (CTB) have built- 
in checks for miscalibration. Real-time hardware bias checking is used to verify that 
scanner calibration is maintained during the scanning process. Additional checks, both 
programmatic and procedural, are implemented by CTB to reinforce the built-in hardware 
checks and to ensure optimal scanner setup. Operations follow standard procedures for 
scanner calibration, including the recalibration of scanning machines every 5000 scans. 
Additionally, FLDOE personnel are present at the scoring contractor's scanning site at the 
beginning of each administration's scanning process to perform an early scan check during 
live scoring to ensure scoring accuracy. Student demographic data and response arrays on 
original answer documents are compared to the electronic scan file to ensure that 
documents are being accurately scanned. 

3. We have two processes in place at the end of the scoring and reporting process that resolve 
any scanning errors missed during the front-end checks. The first process involves an 
automatic manual rescore of tests for Grades 12-Adult Retake students who have failed the 
FCAT Reading or Mathematics test by 25 or fewer scale score points. This ensures that 
incomplete erasures did not affect the outcome for any student who needs a passing score 
for graduation. The second process allows any district to submit a request through a web- 
based system to have student scores reviewed for scoring anomalies or scanning problems. 
This process is well-advertised, available for every test 
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administration, and open to anyone who would like to request it through his or her district 
testing office. 

4. FCAT Test Administration Manuals and test materials contain detailed instructions 
that are read aloud to students prior to all test administrations. The directions (see 
Exhibit 2) clearly instruct students on responding to multiple-choice items, erasing 
completely, completing gridded-response items, and checking their marks to ensure 
they have responded in the correct manner. In addition, FLDOE distributes Sample 
Test Materials to all schools prior to test administration dates which reinforce correct 
answer-marking strategies and give students an opportunity to practice marking 
answers in the correct manner. 

Exhibit 3, prepared by our administration contractor for the 2008 FCAT, shows the result of 
researching two answer documents that were cited as problematic by the auditors. This 
document clearly confirms that the scanner accurately decoded the bubbles on the documents. 

The rules for decoding marks on answer documents are shown in Exhibit 1. The minimum intensity 
level for a bubble to be picked up is 5 out of the possible 15 intensity levels. Many erasures meet the 
threshold of intensity level 5 or higher because students do not erase completely. The threshold is set 
at level 5 in the best interest of students. If the threshold were set lower than level 5, more erasures 
would be decoded as intended marks. If it were set higher than level 5, fewer intended lighter marks 
would be counted as valid marks. 

Scanners are programmed to choose the darkest mark whenever two answer bubbles are darkened in 
the same multiple-choice item or in the same column in a gridded-response item if the intensity levels 
of the two marks are two or more intensity levels apart. In the case of gridded-response items, if an 
erasure is the only bubble gridded in a specific column, it will be picked up as a valid mark if it meets 
the intensity level thresholds described in Exhibit 1 . 

Recommendations from the Auditors 

Regarding Recommendation 1.1: 

1 . 1 Ensure the contractor is correctly setting the Technology Intensity Calibration Algorithm 
to capture students ' gridded responses in the scanner. For responses that are manually entered, have 
a second verification of the entry to ensure the gridded-response items are captured correctly. 

As indicated in Exhibit 1, we require that scanners be calibrated after every 5000 sheets. This entails 
running a calibration sheet through the scanner to ensure that the same level of sensitivity is used 
when all documents are scanned. Our current scoring contractor, Pearson, will be required to fulfill 
this requirement. As an additional assurance, we will require that the contractor produce a report to 
show the frequency of the scanner calibrations. This will provide documentation validating that 
required calibration is completed and guarantee the consistency of the scanning process across all 
documents. 
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While issues regarding manually entered responses are not referenced in Finding No. 1, we understand 
the auditors' recommendation in that there are potential quality control issues with key entry data. 
Standard procedures for responses entered manually require contractor staff to key enter student 
responses when scanning errors occur. Our Scoring Specifications and contractor procedures then 
require subsequent verification of the entry. When FLDOE staff are onsite, this practice is among 
those monitored for compliance. 

Regarding Recommendation 1.2 

1.2 Implement procedures to test a sample of the gridded responses during live scoring to 
ensure students ' gridded responses are accurately scanned. 

The Department has procedures in place to perform this task. Prior to each administration, we send a 
team to the scanning/scoring site to verify that the early scanning of documents is meeting our 
standards. We check the scanning accuracy of documents from at least four districts for all subjects 
and grade levels. Gridded responses, multiple marks, blank answers, and student demographic 
information are reviewed for accuracy. We print documents based on the electronic record provided 
by the scan and/or post-edit files and compare these printed documents to the students' original 
answer documents submitted for scoring. 

EXHIBIT 1: Excerpt from the Scoring Operations Specifications for Spring 
2008 FCAT 

Scanning and document processing 

CTB will scan the FCAT answer documents on 5000i scanners using well established scanning rules 
chosen for Florida. The scanners will pick up carbon marks made by the student on each page. These 
marks will be resolved into valid marks or invalid marks. Invalid marks are classified as blanks or 
multiple-marks. The minimum threshold for a valid mark is set to CTB's standard intensity level 5. 

D.I Intensity Resolution Algorithm 

The Florida Department of Education (FLDOE) has requested a customized solution for resolving 
multiple marks on selected response items in scannable documents. 
CTB Technology defined logic: 

The following defines CTB's understanding of the algorithm: 

• A carbon mark of intensity 4 or lower is considered No Mark. 

• A carbon mark with intensity 5-15 is considered a valid mark. 
Excerpt form the Technology Appendix to the Scoring Specifications: 

The software applies the following logic to resolve multiple marks for all scanned FCAT documents. 

• Rule 1 : Marks in the No Mark range are ignored. 

• Rule 2: If there is no bubbled mark, the item is considered as a Blank. 
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• Rule 3: If there is one bubbled mark, the item is considered as a Single Mark. 

If there are two or more bubbled marks then the darkest two marks are considered in the following 
rules: 

• Rule 4: If the difference in the two intensities is equal to or greater than 2 (e.g., [5, 7] or [13, 
15] or [5, 8]) then the item is considered as a Single Mark and resolved to the bubble with the 
higher intensity. 

• Rule 5: If the difference in the two intensities is less than 2 (e.g., [5, 6], [13, 14], [6, 7]) then 
the item is considered as a Double/Multiple Mark. 

• Refer to the table below for examples. 

Note: 

• Documents that must be updated or inserted manually cannot be included in this analysis. 
This would include, but not be limited to, documents that are folded, crumpled, bent, stapled, 
or torn. 

Operations Process for key entry: 

o If there is a multiple mark, and the clerk can determine which mark the student 
intended to make, they will key enter just that mark. 

o If there are multiple marks for an item and the clerk can not determine which mark 
the student intended to make, they will key enter "M" (for multiple mark). The "M" is 
later converted to the normal multiple mark symbol by the system. 

The following table shows examples of the application logic that resolves multiple marks for all 
scanned FCAT documents: 



Final Report 
ED-OIG/A04I0043 



Page 29 of 36 



Multiple Marks Resolution Algorithm (Examples) 



Highest intensity response 
choice 


Second highest 
intensity response 
choices 


Valid response and 
Erased response 


Scoring Comments 


15 


15, 14 


Not counted for erasure 


Considered multiple mark 


15 


13, 12, 
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Considered as erasure 


14 
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uui i oiuci cu 1 1 1 u i u kJ i c 1 1 1 cii r\ 


14 
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intensity 
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13 
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Considered as erasure 


5 
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5 


3 
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Considered as erasure 
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EXHIBIT 2: Directions for students 
Directions for Completing the Response Grid 

Note: These directions are located in Sample Test Materials, Test Administration Manual scripts read 
to students, and in student test documents for students to reference during testing. 

1 . Work the problem and find an answer. 

2. Write your answer in the answer boxes at the top of the grid. 

• Print your answer with the first digit in the left answer box, OR with the last digit 
in the right answer box. 

• Print only one digit or symbol in each answer box. Do NOT leave a blank answer 
box in the middle of an answer. 

• Be sure to write a decimal point or fraction bar in the answer box if it is part of 
the answer. 

3. Fill in a bubble under each box in which you wrote your answer. 

• Fill in one and ONLY one bubble for each answer box. Do NOT fill in a bubble 
under an unused answer box. 

• Fill in each bubble by making a solid black mark that completely fills the circle. 

• You MUST fill in the bubbles accurately to receive credit for your answer. 

You may refer to the Directions for Completing the Response Grid at any time while taking the 
mathematics test. 

Additional directions for students printed in test documents 

The third symbol is a pencil and a grid. This symbol appears next to questions that require you to fill 
in your answer on a grid in your Answer Book. Answers may be gridded using several correct 
formats. You MUST fill in the bubbles accurately to receive credit for your answer. 

Excerpt from the test administration script 

Now, read silently as I read aloud the Directions for Completing the Response Grid at the bottom of 
the page. 

{Directions for Completing the Response Grid above are read aloud to students) 

You'll also answer multiple choice questions. For multiple choice questions, choose the best answer 
and, in the space provided in your answer book, fill in the bubble for the answer you choose. If you 
use your eraser, do so gently so you don't tear or rip the paper. 

• Mark only the bubble for the answer you choose. 

• Do not make a mark in any of the other bubbles. 

• Do not circle answers. If you circle the answers instead of gridding the bubbles, answers 
will NOT be scored. 

• When you have completed this session, check through your answers to make sure you have 
filled in only one bubble for each question. 
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FLDOE's Response to USED FINDING NO. 2 - Insufficient Monitoring of Florida Department 

of Education's (FLDOE's) Contractor 

Inadequate Safeguards for Discarding FCAT Student PII 

The FLDOE understands and enforces the protection of Personally Identifiable Information (PII). 
Staff take extraordinary precautions to ensure that PII is treated confidentially and destroyed in a 
secure manner when necessary. Further, we require all entities that contract with the Department, 
including assessment contractors, to similarly protect PII. 

Section IX. of Attachment C in the CTB contract states: 

It is understood and agreed that, in accordance with the Family Educational Rights and Privacy 
Act (FERPA) and the Children's On-line Privacy Protection Act (COPPA) it is the intention of 
the parties that personal information, such as student names, grade level and/or other 
identification of students, including students who may be under the age of thirteen, and other 
data concerning students needed to determine or report test answers, test scores, and/or test 
result analyses (hereinafter, "Personal Data"), will be held in confidence by the parties and, if 
gathered or transmitted on-line, whether directly or indirectly, will be deemed gathered or 
transmitted by or under the authority of the Department, including transmission between the 
computers or servers of or under the control of the Department or the Contractor. With respect 
to such Personal Data, it is agreed that all such Personal Data shall be the property of the 
Department and shall be gathered and transmitted to the Contractor by or under the authority 
of the Department in order to enable the Department to assess the performance of the students 
to whom the test is administered and to carry out the Department's educational or other 
responsibilities under law. 

The fact that a specific CTB employee discarded PII in a non-secure fashion, albeit in a secure area, 
does not merit the conclusion that, "FLDOE did not sufficiently monitor its assessment contractor". 
FLDOE Scoring and Reporting personnel are present at scanning sites during each FCAT 
administration's scoring process to ensure that all aspects of the scanning operation proceed according 
to our contractual agreement and that related requirements are met. While we regret that this one 
incident occurred, we do not think this one instance warrants the audit team's conclusion. 

Non-compliance with Contract Terms 

Change of Subcontractor Without Notification 

The FLDOE disagrees with the audit team's conclusion that CTB's change from Kelly Services to 
Apple One for hiring operations violated contractual requirements. 

Section 5.31 (Contractual Information) of the Request for Proposal 2005-001 states: 
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To the extent that the contractor's proposal included named subcontractors, the contractor shall 
utilize the named subcontractors in the manner prescribed in its proposal. Any change of 
subcontractors must be approved in advance by the Department. In the event of poor 
performance by a subcontractor, the Department reserves the right to direct the contractor to 
replace that subcontractor. 

End Excerpt from the Request for Proposal — 

The following chart of subcontractors contained in CTB's proposal indicates which subcontractors 
were identified to provide the requirements described in the RFP. Please note that neither Kelly 
Services nor Apple One is listed on this chart. CTB senior leadership has explained that they 
considered their long-standing arrangements with these agencies for hiring temporary employees 
(such as scanning staff and handscoring staff) as part of CTB's corporate structure, and were not 
required to list them per the RFP. The Department agrees. It is not unreasonable that CTB requested 
FLDOE approval only for changes to subcontractors specifically identified in the proposal as 
responsible for the actual work associated with the FCAT program. 



Tabte 3. Subcontractors and Services 



Subcontractor 


Products and Services 


Solo Printing (CM BE) 


Printing of Ancillary Materials 


Tech ni -forms (CMBE) 


Printing of Answer Documents 


Von Hoffmann Graphics 


Printing of Test Booklets 


Region IV 


Printing of Braille Materials 


Thomas Reprographics 


Printing of Large Print Materials 


Omnl-Vldeo Productions 


Production of training DVD 


Digital Excellence 


Duplication of DVDs 


Measurement incorporated 


Performance Task Scoring for Writing and Mathematics 


indoff 


Calculators (standard, talking, and large key} 


Four Lakes Colorgraphics Inc. 


Independent Proofing of Ancillary Products 


Carlson Wagonlit Travel (CMBE) 


Travel for Committee Members 


e Metric 


Independent Verification of Calibration, Scoring, and Equating 


Precision Translating Services 


Translation of Reports and And Maries into Spanish and Haitian 
Creole 


Floida Instructional Materials Center 


Independent Proofing and Transcription of Braille Test Materials 



In addition, RFP 2008-07 states: 

A separate chart in the proposal will identify all of the subcontractors proposed to be involved 
in the program, the assessment components they will be involved with, and the specific 
services they will provide. Should changes be proposed in subcontractors after the contract 
award, new subcontractors must be approved by the Department throughout the life of the 



Final Report 
ED-OIG/A04I0043 



Page 33 of 36 



contract. Not less than sixty (60) days in advance of a proposed subcontractor's, start of work 
on program processes or deliverables, the contractor will provide the Department with a 
qualifications and experience summary for review and consideration for acceptance. As part of 
the Annual Report, all subcontractors, their services, and Departmental approval status will be 
listed. 

A detailed description of the subcontracted services to be provided under the ensuing contract 
must be included. All subcontracted services will be documented. Each service (e.g., printing, 
computer operations, test development, psychometric services, legal counsel, quality control, 
reporting, etc.) should be fully described including resource, level of service, scheduling and 
quality commitments. The contractor will identify the proportion of materials, etc., to be 
generated by any and all involved subcontractors. 

FLDOE believes that this more specific language regarding subcontractors will be sufficient to 
monitor our current assessment contractor. In addition, the nature of subcontracted work (i.e., directly 
working on assessment-related activities) is clearly stated. 

No Disaster Recovery Plan 

A Disaster Recovery Plan was not available at the assessment contractor scoring site for review by the 
audit team. We concur that the contractor should have been able to provide the audit team a copy of 
this plan, and have requested written assurance from our current assessment contractor that a Disaster 
Recovery Plan is readily available for review at all scanning sites. Verification of a Disaster Recovery 
Plan will be added to the checklist used by FLDOE when monitoring scoring sites for each test 
administration. 

Insufficient Monitoring of Subcontractors 

FLDOE fulfilled its contractual obligation to review a random 20% sample of the scorer candidates' 
qualifications, and the results are on file at the Test Development Center. A review of these records 
indicates that follow-up was requested for candidates that appeared to not have the required 
qualifications. The subsequent investigations resulted in verification and approval of candidate 
qualifications, or dismissal of the candidates. As a result of these reviews, additional candidate IDs 
were sampled to further ensure a qualified pool. It is possible that a review of the entire pool of 
candidates would have resulted in findings similar to the audit findings, however, a 20% sample was 
agreed upon by FLDOE and CTB in recognition of the critical time factor and human resources 
required to conduct the reviews. 

FLDOE and Test Development Center staff have documented that all contractually-required controls 
were followed. In addition, monitoring occurred that went beyond the contractual requirements. 
Evidence of this is in the documentation previously provided to the audit team detailing the extended 
on-site visits to handscoring sites by FLDOE and TDC staff, and in the detailed notes on daily 
handscoring conference calls among the handscoring sites and FLDOE/ TDC leadership. 
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Inadequate Document Control Procedures 

The assessment contractor complied with all document control procedures called for in the assessment 
contract for the 2008 administrations. We require our assessment contractors to be able to retrieve 
(within a 30 day period) assessment documents from the current year's administration. Documents 
from prior years are required to be archived in a secure storage facility, but there is no requirement 
imposed on the assessment contractor to archive these documents in such a way that retrieval can be 
accomplished within a period of time less than 30 days. While the audit team's experience indicates 
some corporate inefficiencies, FLDOE's needs for document retrieval have been met. 

Recommendations from the Audit Team 

Regarding Recommendation 2.1: 

2.1 Use unique identifiers instead of student social security numbers and dates of birth on 
assessment documents. 

The FLDOE will explore the potential use of alternate identifiers for students. However, at this point, 
we will continue to utilize the process currently in place to ensure timely and accurate reporting of 
student results and maintain stringent security policies and procedures. The current process has 
contributed to the accuracy and power of Florida's nationally recognized longitudinal data system. 

Regarding Recommendation 2.2 

2.2 Ensure that all contractors are aware of proper handling ofPII and include language in 
their contracts to properly address the correct handling procedures related to the disposal ofPII. 

We believe that our current contract with Pearson is clear on this issue in several places. 

Contract 09-658 - Attachment C states: 

IX. The Contractor further warrants that the Contractor shall not disclose to any third party, 
without the express, prior, written approval of the Department, any personally identifiable 
information about any student. This applies to information which came from any record or 
report of a Florida public education institution or from any education record which is subject 
to the Family Educational Rights and Privacy Act, 20 U.S.C. Section 1232g. The terms 
"record a report" and "student" shall have the meanings prescribed in s. 1002.22(2)(c) and (d), 
F.S. The term "educational record" shall have the meaning prescribed in 20 U.S.C. Section 
1232g(a)(4). 

Further, the RFP 2008-17, Section 4.4 addresses the disposition of materials: 

Destruction of secure documents and CBT student response files must be requested in writing 
and authorized by the Department. The contractor and all subcontractors must submit 
certificates of destruction that describe in writing the specific items/files destroyed. 
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It is our opinion that the above excerpts provide clear and appropriate requirements for the handling of 
PII. 

Regarding Recommendation 2.3 

2.3 Monitor the contractor to ensure compliance with contract provisions and include a table 
of penalties in the contract for non-compliance with contractual requirements. 

FLDOE has several quality assurance measures including Performance Metrics to hold our contractor 
accountable for contractual obligations. If requirements are not met, payment is withheld as stated in 
the following excerpt from Contract #09-658 - Contract Attachment C (FLDOE' s current assessment 
contract with Pearson). 

EL The Contractor shall prepare an invoice for the amount due and mail it to the Department of 
Education Comptroller after having delivered the products and services required under this 
Contract to the Contract Manager. The invoice shall set forth details sufficient for a proper 
pre-audit and post-audit including, where applicable, the products and services delivered and 
completion dates. Upon receipt of the invoice, the Department of Education Comptroller will 
request confirmation from the Contract Manager that the delivered products and services are 
satisfactory and payment is due. If for any reason they are not satisfactory, payment will be 
withheld until the unsatisfactory condition or conditions are corrected. Upon receipt of the 
Contract Manager's approval, the Department of Education Comptroller shall process each 
invoice in accordance with the provisions of s. 215.422, F.S. 

Regarding Recommendation 2.4 

2.4 Monitor document control procedures at the contractor facilities at least annually. 

The FLDOE has generated additional specific items and processes that will be included in the 
monitoring checklist FLDOE personnel will use when visiting contractor scanning sites. One of these 
monitoring items will be to verify that the procedures used by the contractor to manage and dispose of 
PII and ensure that no PII is disclosed or mishandled. FLDOE staff will be on site to monitor these 
processes during each Fall Retake administration, and twice during the Spring administrations (once 
for Spring Retake and grade 3 assessments, and once for grades 4-11 assessments). 
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FLDOE's Response to USED FINDING NO. 3 - FLDOE's Contractor Delayed Federal Audit 

by Limiting Access to Assessment Documentation 

Notwithstanding the omission of such a clause in 2008 contracts, the FLDOE and its assessment 
contractor did not limit access to documentation required to address the objectives of the Federal 
audit. From the initial site visit by the auditing team to the conclusion of the audit, all requested 
information and data regarding statewide assessments in Florida were provided to the auditing team. 
Multiple strategies were suggested to the audit team on how they might be able to physically check a 
representative sample of original answer documents against the electronic scan file during live 
scanning. However, the audit team requested the two-year-old documents from a secure warehouse 
where processed documents are stored and archived. Due to the nature of this request, the contractor 
may have taken longer to process this request than desired by the audit team, but at no point was 
access denied or limited. 

Recommendations of the Audit Team 

Regarding Recommendation 3.1: 

3.1 Include a Federal audit clause provision in contracts for Department funded programs. 

Our contracts have been modified to include the following federal audit clauses. This language is 
included in the recently executed Pearson/FCAT 2-EOC contract in Attachment A, Section III, D and 
E. 

D. Section XIV. of Attachment C: The Contractor shall grant access to all records pertaining 
to the Contract to the Department's Inspector General, General Counsel and other agency 
representatives, the State Auditor General, the Office of Program Policy and Government 
Accountability, the Chief Financial Officer and Federal Funding Agency (or its authorized 
representative). 

E. Section XV. of Attachment C: The Contractor agrees to permit onsite visits by designated 
Department employees or agents (including any Federal Funding Agency and its 
authorized representative) to conduct audits or to ensure compliance with Section 20.055, 
Florida Statutes. These audits may require Department access to records and data, 
computers and communications devices, and other materials whether owned or operated by 
the Contractor. Access may include, but is not limited to, user level and/or system level 
access to any computing or communications device; access to information (electronic, 
hardcopy, etc) that may be produced, transmitted or stored on the Contractor's equipment 
or premises; access to work areas; and access to interactively monitor and log traffic on the 
Contractor's networks. 

Regarding Recommendation 3.2: 

3.2 Include a table of penalties in the contract for non-compliance with a Federal audit. 

The Department will review the liquidated damages section of our contracts and determine whether or 
not penalties for non-compliance should be included in future contracts. 



